Status:

This documentation applies to version 1.48.853.82 (4. December-2024) of the VdAstrid2/CVdAst2ErrorFix device.

 Contents:

• CVdAst2ErrorFix overview
• Error Fix device
• ASTRID2 error fix scheduler device
• ASTRID2 automation device
• Vd2Astrid
• F0: A2 ring error recovery
• F1: A2 Transfer beamline error recovery
• F2: Very poor A2 beam lifetime which does not go away after an injection

CVdAst2ErrorFix overview

Device design notes - see document(s) in folder

O:\Nat_ISA-Ast2Doc\machine\Documentation\Controls\Automation\ErrorFix

This devices implements error fix routines for ASTRID2.

F0: A2 ring error recovery

An errorFix to recover operation if there is trouble with BMH100IPS, QMH101IPS, QMH102IPS or any of the A2 IFA corrector supply distributors.

Cluster name: EFX01A2RINGERast2

Trigger

    • MachineMode: StatusAst2.MachineMode in OnAutoRecovery mode and
       • Trig if one of the supplies 'Main' is off
         • BMH100IPSast2.Main
         • QMH101IPSast2.Main
         • QMH102IPSast2.Main
       • Trig if SumFailure in any of the A2 distributors

Action

    • Set Decaying
    • Wait 30 s
    • Test that ConSys connections are OK (no network failure)
       • Wait up to 15 minutes for ConSys connections to be OK
    • Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
    • Trigger a AutoOnOffAst2.turnRingOn (which turns "everything" on, inclusive a magnet wash).
    • Wait for AutoOnOffAst2.DeviceBusy going false
    • If AutoOnOffAst2.lstActionSuccess is false, then trigger make another try (go back start)
    • and wait for AutoOnOffAst2.DeviceBusy going false
    • Wait 30 seconds
    • If AutoOnOffAst2.lstActionSuccess Go To Top-up

Action Progress

    0: Inactive
    Trigged - actionProgress is OR between
       1: BMH101 Main off
       2: BMH101 Main off
       4: BMH101 Main off
       8: One of the A2 Corrector distributers has failure
    Recover try in progress (n is try number - 1,2):
        n00xx: Wait (xx seconds left).   *1
        n1xxx: Check connections (xxx seconds waited).   *1
        n5000: 'Turn Ring On' send
        n6xxx: Waiting for 'deviceBusy' FALSE (xxx seconds waited).   *1
        n9000: Timeout while waiting for 'deviceBusy' FALSE
        n9001: ConSys connection failed (invalid value(s)
        n9002: Recovery aborted - machineMode no longer in OnAutoRecovery
    Recovered:
        8xx: Wait after recovered (xx seconds left).   *1

F1: A2 Transfer beamline error recovery

An errorFix to recover operation need an ErrorFix to recover operation if there is trouble with power supplies in the transfer Beamline.

Cluster name: EFX01A2TRNBLERast2

Trigger

    • MachineMode: StatusAst2.MachineMode in OnRecovery mode and
       • Trig if one of the supplies 'Main' is off:
         • SMH111IPUast2.Main
         • FMH161IPUast2.Main
         • FMH111IPUast2.Main
         • FMH121IPUast2.Main
         • QMV211IPSast2.Main
         • QMH211IPSast2.Main
         • QMH212IPSast2.Main
         • QMV212IPSast2.Main
         • BMV211IPSast2.Main
         • BMV212IPSast2.Main
   
If trigged, EFX01A2TRNBLERast2.triggeredBy is set to word with bits set for supplies having main failure (bit numbers in the order of the supplies listed above, i.e. SMH111IPUast2.Main =0x01,FMH161IPUast2.Main =0x02 etc.

Action

    • Test and wait for ConSys connections are OK (no network failure)
    • Wait additional 30 s
    • Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
    • If A2ringErrorRecovery is not running (if running just exit because then the action is done in A2ringErrorRecovery)
       • Set Decaying
       • Read state of AutoOnOffAst2.WashWhenTurningOn
       • Set AutoOnOffAst2.WashWhenTurningOn false
       • Recover try in progress (n is try number - 1,2):
          • Check that MachineMode is still in OnAutoRecovery - if not so, abort action.
          • Trigger a AutoOnOffAst2.turnRingOn (which turns "everything" on, but without the magnet wash).
          • Wait for AutoOnOffAst2.DeviceBusy going false
          • Check AutoOnOffAst2.lstActionsuccess - if ok, recovered.
       • Set back the old state of AutoOnOffAst2.WashWhenTurningOn
    • If recovered
       • Wait 30 seconds
       • Go To Topup

Action Progress

   0: Inactive
   1: Trigged - triggeredBy is OR between
     0x01 SMH111IPUast2.Main off
     0x02 FMH161IPUast2.Main off
     0x04 FMH111IPUast2.Main off
     0x08 FMH121IPUast2.Main off
     0x10 QMV211IPSast2.Main off
     0x20 QMH211IPSast2.Main off
     0x40 QMH212IPSast2.Main off
     0x80 QMV212IPSast2.Main off
     0x100 BMV211IPSast2.Main off
     0x200 BMV212IPSast2.Main off
   2: Waiting for ConSys connections ok
   1xx: Wait (xx seconds left).   *1
   1000: A2 Ring Recovery is active - no action needed - already being done by A2 Ring Recovery.
   3: Recovery aborted, MachineMode is no longer in OnAutoRecovery.
   4: Set Decaying
   Recover try in progress (n is try number - 1,2):
     n1000: 'Turn Ring On' send
     n2xxx: Waiting for 'deviceBusy' FALSE (xxx seconds waited).   *1
     n3000: Waiting for 'deviceBusy' timeout
     n4000: No success, not all mains on
     n5000: No success, lstActionsuccess false
     n6000: Recovery aborted - MachineMode no longer in OnAutoRecovery
     n6001: Recovery aborted while waiting for turn ring on finisg - MachineMode no longer in OnAutoRecovery
     n9000: Try n Turn ring on successful
   8xx: Waiting for things to stabilize (xx seconds left).   *1
   5: Restore BeamMode

F2: Very poor A2 beam lifetime which does not go away after an injection

One time recover try of very poor A2 beam lifetime which does not go away after an injection (dust particle?).

Cluster name: EFX100A2POORLFTMast2

Trigger

    • ErrorFix is On
    • StatusAst2.MachineMode in OnRecovery mode
       • AND (1 <= A2BeamCurrent.normal <=20)
       • AND (A2BeamCurrent.tau20s < 0.1 h )
       • AND (InjectionAst2.Efficiency >= 25%)
    • Condition must be true for (at least) 1 minute (we need to have at least two injections - to verify transfer efficiency is good)

Action

Kick out beam (insert viewer MVS141MNPast2.ctrl) and refill (Scheduler Request Beam-mode restore).
This is a one time fix, as there could be other reasons for very poor A2 beam lifetime. State should go to 'failNoBlock' in case of failure.

Action Progress

   0: Inactive
   1..60: Trigger condition fulfilled count - action is trigged when the counter reaches 60
     1000: Action trigged, Viewer In
     2000: Viewer Out
     3xxx: Waiting for scheduler restore beam mode (filling),xxx seconds waited
     4xxx: Beam mode is filling - Waiting 3 minutes, xxx seconds left
     5000: Recovered, filling ok
     6000: Not recovered, Filling conditions not ok
     7000: Not recovered, Beam mode not in filling

*1

ActionProgress: When counting up, only the first and the last value are added to the read/write histories in order to make the histories more readable.

Last Modified 05 December 2024